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Abstract 

In most compressive sensing problems £i norm is used during the signal reconstruction process. In tliis article the use of entropy 
functional is proposed to approximate the £i norm. A modified version of the entropy functional is continuous, differentiable and 
convex. Therefore, it is possible to construct globally convergent iterative algorithms using Bregman's row action D-projection 
method for compressive sensing applications. Simulation examples are presented. 
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I. Introduction 

Nyquist-Shannon sampling theorem |[T1 is one of the fundamental theorems in signal processing literature. As it is well 
known, it specifies the conditions for perfect reconstruction of a continuous signal from its samples. If a signal is sampled 
with a rate at least two times its bandwidth, it can be perfectly reconstructed from its samples. However in many applications 
of signal processing including waveform compression, perfect reconstruction is not necessary. 

The most common method used in compression applications is the transform coding. The signal x.[n] is transformed into 
another domain defined by the transformation matrix ip . The transformation procedure is simply finding the inner product of 
the signal x[n] with the rows tpi of the transformation matrix ^ as follows 

s, =< x,V', >, i = l,2,...,iV, (1) 

where x is a column vector, whose entries are samples of the signal x[n] . The digital signal x.[n] can be reconstructed from 
its transform coefficients Si as follows; 

N 

X — Si-ipi or X = ijj.s (2) 

where s is a vector containing the transform domain coefficients, Si. The basic idea in digital waveform coding is that the signal 
should be approximately reconstructed from only a few of its non-zero transform coefficients. In most cases including JPEG 
image coding standard, the transform matrix is chosen such that the new signal s is easily representable in the transform 
domain with a small number of coefficients. A signal x is compressible, if it has a few large valued Si coefficients in the 
transform domain and the rest of the coefficients are either zeros or very small valued. 

In compressive sensing framework the signal is assumed to be a K-Sparse signal in a transformation domain such as wavelet 
domain or DCT domain. A signal with length N is K-Sparse, if it has K non-zero and (A^ — K) zero coefficients in a transform 
domain. The case of interest in CS problems is when K « N i.e., sparse in the transform domain. 
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Fig. 1. Entropy functional g(v) (— ), \v\ (o) and tlie Euclidean cost function (x) 

The CS theory introduced in |l2l-||5| provides answers to the question of reconstructing a signal from its compressed 
measurements y, which is defined as follows; 

y = 0.x = (j).ip.s = 9.S (3) 

where is the M x N measurement matrix where M << N. The reconstruction of the original signal x from its compressed 
measurements y cannot be achieved by simple matrix inversion or inverse transformation techniques. A sparse solution can 
be obtained by solving the following optimization problem; 

Sp = a7'5mm||s||o such that 9.s = y. (4) 

However this problem is a NP-complete optimization problem therefore its solution can not be found easily. It is also shown 
in 12], Q that, it is possible to construct the matrix from random numbers which are iid Gaussian random variables and 
choose the number of measurements as cKlog{N / K) < M ^ N to satisfy the reconstruction conditions defined in JS] and 
[01 . With this choice of the measurement matrix, the optimization problem (|4]l can be approximated by Ix norm minimization 
as follows, 

Sp — argniin\\s\\i such that 0.s — y (5) 

Entropy functional g{v) — vlogv is used to approximately solve some £i optimization problems and linear programming 
problems in signal and image reconstruction problems by Bregman (|6l and others ifTSl . lfT4l . ifTSl . ||20| and |fT6l . In this article 
we propose the use of entropy functional as an alternative way to approximate the CS problem. In Figure [1] plots of the 
different cost functions including the shifted entropy function 

giv) = {\v\ + -)\og{\v\ + -) + -, (6) 
e e e 

absolute value g{v) = \v\ and g{v) = are shown. The shifted entropy functional (|6]) is convex, continuous and differentiable, 
and it slowly increases compared to g(y) = because log(u) is much smaller than v for high v values as seen in Figure 
1. Bregman also developed iterative row-action methods to solve the global optimization problem by successive local D- 
Projections. In each iteration step a D-projection, which is a generalized version of the orthogonal projection is performed 
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onto a hyperplane representing a row of the constraint matrix 6. In (6T| Bregman proved that the proposed D-Projection based 
iterative method is guaranteed to converge to the global minimum regardless of the initial estimate. 

An interesting interpretation of the row-action approach is that it provides a solution to the on-line CS problem. Each new 
measurement of the signal adds a row to the matrix 9. In the iterative row-action method a D-projection is performed to the new 
hyperplane formed by the new measurement. In this way, the currently available solution is updated without solving the entire 
CS problem. The new solution can be further updated using past measurements or new measurements in an iterative manner 
by performing other D-Projections. Therefore, it is possible to develop a real-time on-line CS method using the proposed 
approach. 

The paper is organized as follows. In Section |II] we review the D-Projection concept and define the modified entropy 
functional and related D-Projections. We generalized the entropy function based convex optimization method introduced by 
Bregman because the ordinary entropy function is defined only for positive real numbers. On the other hand, transform domain 
coefficients can be both positive and negative. The iterative CS algorithm is explained in Section ITl-AI We provide experimental 
results in Section |III1 

II. D-Projection based Algorithm 

The lo and £i norm based cost functions (|4|i and (|5]l used in compressive sensing problems are not differentiable everywhere. 
In this article we use entropy functional based cost functions to find approximate solutions to the inverse problems defined 
in llJl, [3]. Bregman developed convex optimization algorithms in 1960's and his algorithms are widely used in many signal 
reconstruction and inverse problems ||9l- lfT5l , lfT8l - ll22l . 

Assume that the original signal x can be represented hy ?l K sparse length-N vector s in a transform domain characterized by 
the transform matrix ^jj. In CS problems the original signal x is not available. However M measurements y ~ [j/i, 2/2, VmY" — 
ipx of the original signal is observable via the measurement matrix 0, and the relation between y and s are described in Eq. 
©. 

Bregman's method provides globally convergent iterative algorithms for problems with convex, continuous and differentiable 
cost functionals g{.): 

mm g{s) (7) 

such that 

e,.s^y, for z = l,2,...,M. (8) 

where 9i is the i-th row of the matrix 9. Each equation in ^ represents a hyperplane Hi in , which are closed and convex 
sets in . In Bregman's method the iterative reconstruction algorithm starts with an arbitrary initial estimate and successive 
D-projections are performed onto the hyperplanes Hi, i = 1, 2, M in each step of the iterative algorithm. 

The D-projection onto a closed and convex set is a generalized version of the orthogonal projection onto a convex set [6|. 
Let So be arbitrary vector in R^ . Its D-Projection Sp onto a closed convex set C with respect to a cost functional g(s) is 



defined as follows 



Sp = arg min D{s, Sq) (9) 

sec 



where 

D(s, So) .g(so) - .g(s)- < \yg{s),so - s) > (10) 

In CS problems, we have M hyperplanes Hi : 6i.s = yi for i — 1,2, M. For each hyperplane Hi, the the D-projection 
(|9]l is equivalent to 

V5(sp) = V5(so) + (11) 

dfSp = (12) 

where A is the Lagrange multiplier. As pointed above the D-projection is a generalization of the orthogonal projection. When 
the cost functional is the Euclidean cost functional ^(s) = s{n)'^ the distance D{sx, S2) becomes the £2 norm of difference 
vector (si — S2), and the D-projection simply becomes the well-known orthogonal projection onto a hyperplane. 
The orthogonal projection of an arbitrary vector Sq = [so[l], so[2], so[-Af]] onto the hyperplane Hi is given by 

Sp{n) = so{n) + X9^{n),n = 1,2, ...,N (13) 

where 6i [n) is the n-th entry of the vector 9i and the Lagrange multiplier A is given by, 

^ y» -En^i'So(»)g(»,n) 

When the cost functional is the entropy functional g{s) — s{n) log(s(n)), the D-projection onto the hyperplane Hi leads 
to the following equations 

Spin) = So(n).e(^-^'("», n^l,2, N (15) 

where the Lagrange multiplier A is obtained by inserting (fTSl l into the following hyperplane equation: 

0iS = yi (16) 

because the D-projection Sp must be on the hyperplane Hi. The above set of equations are used in signal reconstruction from 
Fourier Transform samples ||20l and the tomographic reconstruction problem |fT3l . The entropy functional is defined only for 
positive real numbers. In CS problems entries of vector s can take both positive and negative values. We modify the entropy 
functional and extend it to negative real numbers as follows: 

^ 1 1 

minV(|s,| + -).(/05(|s,| + -)) s.t. O.s^y. (17) 
^-^ e e 

where subscript e represents the term entropy. The continuous cost functional ge{s) satisfies the following conditions, 
(i) |f^(0) = 0, i = l,2,...,7Vand 
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Fig. 2. Geometric interpretation of tlie entropic projection method: Sparse representation s; corresponding to decision functions at each iteration are updated 
so as to satisfy the hyperplane equations defined by the measurements j/^ and the measurement vector 9i. Lines in the figure represent hyperplanes in R^. 
Sparse representation vector Si converges to the intersection of the hyperplanes. Notice that D-projections are not orthogonal projections. 

(ii) ge is strictly convex and continuously differentiable. 

On the other hand, the £i norm is unfortunately not a globally smooth function therefore it can be solved using non 
differentiable minimization techniques such as sub-gradient methods Q. Another way of approximating the ii penalty function 
using an entropic functional is available in (SJ. 

To obtain the D-projection of Sq onto a hyperplane Hi with respect to the entropic cost functional ( fTTI l. we need to minimize 
the generalized distance D{s, So) between sq and the hyperplane Hi: 

D{S, So) = ge(So) - ge(s)+ < Vge(s), S - Sq > (18) 

with the condition that 9iS = yi. Using (fTZt . the values of s can be obtained as; 

s{n) = {so{n) + sgn{s{n)).e-^).e^'3n{4'''>).\.9.{n)) _ (^sgn{s{n)).e-^), n = 1, 2, N. (19) 

where A is the Lagrange multipUer. 

The D-projection vector and the s* satisfies the set of equations ( fT9] l. and the hyperplane equation Hi : 9i.s = yi. 

A. Iterative Reconstruction Algorithm 

The global convex optimization problem defined in fTl\ is solved by performing successive local D-projections onto 
hyperplanes defined by the rows of the matrix 9. 

The iterations start with an arbitrary initial estimate Sq. This vector is D-projected onto the hyperplane Hi and Si is obtained. 
The iterate Si is projected onto the next hyperplane H2 (see Figure |2]l... N — 1"* estimate SAr_i is D-projected onto Hn and 
Sat is obtained. In this way the first iteration cycle is completed. The vector sjv is then projected onto the hyperplane Hi and 
SAr_|_i is obtained etc. Bregman proved that Si defined in ( [TtI i converges to the solution of the optimization problem. 

III. Experimental Results 

For the validation and testing of the entropic minimization method, two experiments with two different signals are carried 
out. The cusp signal (Figure |3]l, which has 1024 samples, and the random sparse (Figured signal, which has 128 samples, 
are used. The cusp signal is 5 = 72 sparse in DCT domain and the random signal has four non-zero samples. 
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Fig. 3. The cusp signal with = 1024 samples 




Fig. 4. Random S=4 sparse signal with A'^ = 128 samples 

The measurement matrices (p are chosen as Gaussian random matrices. In the experiments M ~ 2S and M = lOS 
measurements are taken from the cusp signal and M = 6S and M = lOS measurements are taken from the random signal. 
The original signals are reconstructed from those measurements. The reconstructed signals using the iterative method using 
entropy based cost functional are shown in Figures [5(a)| |5(b)[ |6(a)[ and |6(b)| The reconstructed signals using the pseudo-inverse 
of the 9 matrix are shown in Figures |7(a)| and |7(b)| respectively. 

IV. Conclusion and Future Work 

In this article we present; (i) the entropy based cost function for solving the CS problems, and (ii) an iterative row-action 
method to solve the CS problem. The entropy based cost functional estimates the £i norm. Both the theoretical results given 
in Section |ll] and the experimental results given in Section |lll] indicate that the entropy based cost function and the iterative 
row-action method can be used for reconstructing a sparse signal from its measurements. 

It is also shown that the row action methods provide a solution to the on-line CS problem. The reconstruction result can be 
updated on-line according to the new measurements without solving the entire optimization problem again in real time. 

We will compare this entropy functional approach with other methods available including the TV approach ||231 . Il24l 
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(a) TV = 1024 length cusp signal reconstructed from 25 = 144 measurements 
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(b) A'^ = 1024 length cusp signal reconstructed from lOS = 720 measurements 

Fig. 5. The cusp signal with 1024 samples reconstructed from _A/ = 25" (a) and M = lOS (b) measurements using the iterative, entropy functional based 
method. 
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(a) TV = 128 length random sparse signal reconstructed from 65 = 24 measurements 
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(b) TV = 128 length random sparse signal reconstructed from 105" = 40 measurements 

Fig. 6. Random sparse signal with 128 samples is reconstructed from (a) M = 65 and (b) TV/ = 105 measurements using the iterative, entropy functional 
based method. 
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(a) The DCT of the cusp signal reconstructed from its measurements using pseudo-inversion v.s. the DCT of the original cusp signal. 
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(b) Random sparse signal reconstructed from its measurements using pseudo-inversion v.s. the original signal. 
Fig. 7. The DCT of the cusp signal and the random sparse signal reconstructed from their measurements using pseudo-inversion. 
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